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Submitted  to  the  Department  of  Electrical  Engineering 
and  Computer  Science  on  May  25 » 1979  in  partial  fulfillment 
of  the  requirements  for  the  degree  of  Master  of  Science. 

ABSTRACT 

When  independent  sources  use  a common  broadcast  channel, 
interference  between  the  sources  can  cause  messages  to  be 
lost.  A common  recovery  procedure  is  to  wait  a random  length 
of  time  before  attempting  retransmission.  Capetanakis  has 
devised  a tree  algorithm  for  determining  when  users  should 
retransmit.  This  algorithm  has  better  throughput,  delay  and 
stability  characteristics  than  the  random  retransmission 
procedure.  Galla,ger  and  Humblet  have  modified  it  to  have 
higher  throughput  and  the  property  that  messages  are  trans- 
mitted in  the  order  of  generation. 

In  this  thesis,  the  general  form  of  a first-come  first- 
served  contention  resolution  algorithm  is  given.  A sub- 
class of  FCFS  algorithms,  called  single  interval  algorithms, 
are  analyzed  using  Markovian  decision  theory.  It  is  seen 
that,  if  the  horizon  is  infinite,  Humblet' s algorithm  has 
the  maximum  throughput  for  this  sub-class.  However,  if  the 
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system  is  shut  down  after  a finite  number  of  transmissions, 
then  an  extended  version  of  his  algorithm  is  optimal. 
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I.  INTRODUCTION 


In  this  thesis  we  present  the  most  general  form  for  a 
contention  resolution  algorithm  that  is  first-come  first- 
served.  We  model  the  algorithm  as  a Markov  process  with 
state  transition  rewards  and  use  Markovian  decision  theory 
to  analyze  the  maximum  throughput,  or  capacity,  for  several 
restricted  forms  of  the  general  algorithm. 

In  this  chapter,  we  present  the  system  model  used. 
Existing  algorithms  are  described  and  evaluated  qualitative 
ly.  The  general  first-come  first-served  algorithm  is  given 
and  the  sub-class  cf  FCFS  algorithms  which  we  will  analyze 
described.  We  complete  the  chapter  with  an  outline  of  the 
thesis . 

1.1  Description  cf  the  Problem 

We  are  concerned  with  a large  number  of  bursty  users 
trying  to  communicate  with  a central  facility  on  a common 
channel.  The  traditional  methods  cf  multi-accessing,  time 
division  multi-accessing  (TDM.A)  and  frequency  division 
multi-accessing  (FDM.A)  are  not  suitable.  Since  the  number 
of  users  is  assumed  to  be  very  large,  FDKA  is  prohibitive 
in  hardware  costs,  and  since  the  users  are  bursty  as  well, 
both  FIMA  and  TIM.A  require  trade-offs  between  delay  and 
bandwidth  that  result  in  either  unacceptable  delays  or 
inefficient  use  of  channel  capacity.  In  this  situation 
schemes  have  been  proposed  in  which  the  users  access  the 


channel  randomly  and,  when  interference  causes  messages  to  be 
lost,  a contention  resolution  algorithm  is  applied  to  deter- 
mine when  the  users  should  retransmit. 

There  are  several  criteria  for  judging  such  a scheme. 

It  should  be  stable;  that  is,  the  expected  number  of  users 
waiting  to  transmit  should  remain  small.  It  should  allow 
messages  to  be  transmitted  with  low  average  delay.  And  it 
should  have  a high  throughput  (percentage  of  capacity  used 
to  transmit  successfully).  Algorithms  which  meet  these  three 
requirements  already  exist  and  we  will  review  some  of  them 
below. 

1.2  Source  and  Channel  Models 

Before  discussing  the  algorithms,  we  will  describe 
the  source  and  channel  models  used.  We  consider  a central 
facility  with  an  infinite  number  of  users,  also  called 
sources.  The  sources  generate  messages  of  fixed  length, 
referred  to  as  packets.  Eecause  the  number  of  users  is 
infinite,  each  source  is  assumed  to  generate  only  one  mes- 
sage. A slot  is  the  length  of  time  required  to  transmit 
a packet.  A source  may  generate  a packet  at  any  time,  but 
may  transmit  only  at  the  times  specified  by  the  algorithm 
being  used.  The  generation  times  are  assumed  independent 
and  form  a Poisson  point  process  with  (global)  arrival  rate 
A,  measured  in  packets  per  slot.  When  two  or  more  sources 
transmit  so  that  their  messages  overlap  it  is  assumed  that 
all  the  messages  are  lost  and  require  retransmission.  We 
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further  assume  that  if  only  one  source  transmits,  the  message 


is  received  perfectly.  This  assumption  is  realistic  as  long 
as  the  probability  of  collision  is  much  greater  than  the 
probability  of  message  error,  which  will  be  true  if  the 
signal-to-noise  ratio  is  high  or  forward  error  correction 
is  used.  Finally,  we  assume  that  all  the  users  are  able  to 
listen  to  the  channel  and  determine  instantaneously  whether 
0,1  or  more  than  one  message  were  just  sent.  Alternatively, 
this  information  can  be  provided  by  a feedback  channel. 

We  are  concerned  with  maximizing  the  throughput  of  the 
channel.  We  define  the  capacity,  > * , of  the  conflict  re- 
solution algorithm  to  be  the  supremum  over  all  arrival  rates 
X such  that  the  expected  number  of  messages  generated  but 
not  yet  successfully  transmitted  remains  bounded.  If  ' 
is  less  than  the  capacity,  then  the  throughput  is  " and 
the  system  is  stable.  If  a exceeds  the  capacity,  then  the 
throughput  is  at  most  X*. 

1.3  Access  Strategies 

In  this  section,  we  discuss  a number  of  schemes  for 
accessing  the  broadcast  channel:  Alena  and  its  modifications, 
Capetanakis'  tree  algorithm,  and  a first-come  first-served 
algorithm  proposed  by  Gallager  and  modified  by  Humble t. 

1.3.1  Alcha 

Our  model  is  a fairly  accurate  repress  station  cf  the 
Aloha  system,  a radio  packet  computer  communication  network 
developed  at  the  University  of  Hawaii.  In  this  system,  when 


10 
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a conflict  occurs,  contending  sources  wait  a random  length  of 
time  before  retransmitting.  This  scheme  has  been  quoted  as 
having  a maximum  throughput  of  l/2e  under  the  (false)  assump- 
tion that  the  total  traffic  is  Poisson  (1).  A modification 
to  this  scheme,  called  slotted  Aloha,  has  been  proposed  in 
which  users  are  constrained  to  start  transmission  at  discrete 
times,  at  one  slot  intervals  (2).  This  increases  the  "through- 

i 

put"  to  l/e.  Both  schemes,  however,  can  be  shown  to  be  un- 
stable when  the  number  of  sources  is  infinite  (3),  although 
this  has  not  been  a problem  in  practice,  due  to  the  small 
number  of  actual  users. 

Another  scheme,  known  as  reservation  Aloha,  has  been 
proposed  in  which  tv/o  virtual  channels  are  used  (4).  A 
user  requests  channel  capacity  ever  the  reservation  channel 
and  dynamic  TD1A  is  used  to  divide  the  other  channel  among 
those  who  have  requested  it.  This  is  only  useful  when  the 
sources  send  messages  that  are  much  longer  than  the  request 
packets,  and  since  the  request  channel  is  still  accessed 
randomly,  it  can  suffer  from  poor  stability  and  low  through- 
put. 

Another  Aloha  scheme,  carrier  sense  multiple  access  (5). 
requires  users  to  "listen"  to  the  channel  before  transmitting 
to  see  if  there  is  a transmission  already  in  progress.  It 
is  only  useful  when  the  sum  of  the  propagation  time  between  ( 

users  and  the  time  necessary  to  detect  that  the  channel  is  * 

t 

idle  is  much  smaller  than  a slot.  This  means  that  the  users 

kil 


ii 


must  be  relatively  close  together.  It  also  has  stability 
problems,  since  many  users  may  attempt  to  transmit  sim- 
ultaneously when  a packet  terminates. 

1.3.2  The  Tree  Algorithm 

In  1977 i Capetanakis  introduced  a tree  algorithm  for 
resolving  conflicts  (6).  In  this  scheme  the  sources  are 
each  assigned  an  address  which  specifies  their  location  on 
a binary  tree.  When  a conflict  occurs,  the  tree  is  divided 
into  two  halves  and  the  sources  in  one  half  attempt  retrans- 
mission first.  The  sources  in  the  second  half  must  await 
retransmission  until  all  conflicts  in  the  first  half  are 
resolved.  If  further  conflicts  occur  upon  division,  the 
procedure  is  repeated.  The  period  of  time  between  the  slot 
in  which  a conflict  first  occurs  until  it  is  finally  re- 
solved is  referred  to  as  an  epoch.  Any  new  messages  arrivir. 
during  this  resolution  process  must  wait  until  the  end  of 
the  epoch  before  attempting  transmission. 

He  then  generalized  his  algorithm  to  allow  the  root- 
node  of  the  tree  to  be  of  degree  greater  than  two,  choosing 
it  to  minimise  the  expected  epoch  length.  In  addition  to 
having  good  delay  characteristics,  this  scheme  is  stable 
for  3 less  than  .430. 

1.3.3  7-allamer's  Algorithm 

Gallager  (7)  has  since  modified  this  algorithm  to  cr.e 
which  is  first-come  first-served  (FCFS),  maintains  stabilit 
and  good  delay,  and  has  a capacity  of  .48711.  By  first-come 
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first-served,  we  mean  that  no  message  is  ever  successfully 
transmitted  before  another  message  with  an  earlier  generation 
time.  In  Gallager's  algorithm,  each  source  in  the  system 
maintains  a record  of  the  values  of  three  variables:  the 
time  at  which  the  source  generates  a packet  to  be  transmitted, 
and  two  system  parameters,  a lag,  d,  and  an  interval  length, 
t.  These  parameters  represent  the  state  of  the  system  and 
are  constantly  being  updated.  At  the  start  of  each  new  slot, 
only  those  sources  will  send  which  generated  messages  during 
the  time  interval  of  length  t (measured  in  slots)  v/hich 
began  d slots  past.  This  interval  will  be  referred  to  as  the 
transmission  interval,  and  if  a conflict  occurs  in  it,  it  is 
called  a conflict  interval. 

The  operation  of  this  scheme  and  the  updating  of  d and 
t are  most  easily  shewn  by  an  example.  In  figure  1,  the 
arrows  along  the  time  axes  indicate  times  at  which  sources 
have  generates,  messages  for  transmission.  The  heavy  vertical 
bars  at  the  right  of  each  time  line  show  the  location  cf  the 
present- slot , and  the  number  between  the  lines  indicates 
whether  the  channel  sees  no  message,  cne  message  or  a con- 
flict. VJe  assume  in  (1.1)  that  we  begin  w ith  some  known 
values  of  t and  d,  and  we  observe  ho w they  are  changed.  Since 
only  one  message  was  generated  in  the  interval  of  length  T , 
beginning  d slots  ago,  there  is  no  conflict  ir.  the  present 
slot.  Hence  the  lag  parameter  is  updated  according  to: 

^new  - dold  + * Told  * 
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In  this  case,  we  know  nothing  of  the  statistics  of  the 

message  generation  times  in  the  remaining  interval  up  to  the 

present  time.  We  call  such  an  interval  the  waiting  interval, 

since  all  messages  generated  during  that  interval  may  be 

thought  of  as  awaiting  their  first  transmission.  The  set 

of  all  messages  waiting  to  transmit,  whether  in  the  waiting 

interval  or  not,  is  called  the  queue.  We  assume,  then, 

that  there  exists  some  optimal  interval  of  length  x , 

0 

which  we  should  use.  (The  value  of  x^  will  be  determined 

o 

later.)  Hence  we  set  x = T0*  In  (1.2a)  we  find  that  the 
transmission  interval  contains  a conflict.  The  algorithm 
states  that  in  mhe  next  slot  we  should  examine  the  contents 
of  the  first  half  of  the  interval.  Hence 


d = d , , + 1 

new  cld 

T - X /2 

new  old7 

In  (1.2b)  we  find  that  the  first  half  of  the  interval  con- 
tained no  messages  and  so  the  conflict  must  be  in  the  second 
half.  Since  there  is  a known  conflict  in  the  second  half,  we 
use  only  the  third  quarter  as  our  new  transmission  interval: 

d =d,.  + l-  x., 
new  old  old 

T = T , J2  . 
new  old' 

In  (1.2c)  we  again  find  a conflict.  As  a consequence  cf  this 
we  see  that  nothing  is  known  about  the  statistics  of  the 
arrival  times  in  the  final  auarter,  and  we  no  longer  regard 


it  as  part  of  the  present  conflict  interval.  That  is,  we 
return  it  to  the  waiting  interval.  We  divide  the  interval 

15 


again  and  let*. 


d =d,j  + l-T,. 

new  old  old 

T = x . , . 
new  old 

In  (1.2e)  we  find  only  one  message  sent  and  there  are  no 
further  conflicts.  The  epoch  is  over  and  we  begin  again  in 
(1.3)  with: 


dnew  ^old  + x Told 

T = T 

new  0 

Gallager  shows  that  for  the  system  to  be  stable,  \ must 

be  less  than  or  equal  to  .43711  and  that  - satisfies 

0 

At  = 1.266. 


At  the  beginning  of  a new  epoch  it  is  possible  that  the 

waiting  interval  is  smaller  than  t . In  this  case  we  may 

either  transmit  the  arrivals  in  the  waiting  interval  cr  wait 

until  we  can  take  an  interval  of  ler.sth  " . The  first  choice 

0 

will  probably  yield  a shorter  average  delay. 

Gallager' s algorithm  is  much  like  Capetanakis*  in  That 
it  divides  the  remaining  contending  sources  in  half  at  the 
start  of  each  new  slot.  There  are  several  differences,  how- 
ever, which  improve  capacity  and  possibly  delay.  First,  in 
the  case  where  the  first  half  of  a conflict  interval  con- 
tains no  messages,  so  that  there  is  a known  conflict  in  the 
second  half,  the  tree  algorithm  sends  the  messages  generated 
in  the  whole  second  half , while  Gallager' s scheme  send  only 
the  messages  in  the  third  quarter.  Next,  when  a conflict 


is  found  in  the  first  half  of  a conflict  interval,  Gallager's 
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scheme  returns  the  second  half  to  the  waiting  interval,  while 
in  an  analogous  situation,  the  Capetanakis  algorithm  still 
considers  the  second  half  of  the  tree  part  of  the  conflict. 
These  differences  are  probably  the  major  source  of  improve- 
ment. Finally,  Gallager's  scheme  begins  a new  epoch  with  an 
interval  whose  length  has  been  optimally  determined,  whereas 
the  Capetanakis  algorithm  is  constrained  to  choose  an  integer 
for  the  degree  of  the  new  optimum  tree's  root  node. 

1.3.4  Humble t's  Algorithm 

Humble t (8)  considered  Gallager's  algorithm  and  pro- 
posed the  following  modifications.  First,  the  optimal  div- 
ision of  a conflict  interval  is  not  always  in  equal  parts. 
Instead  it  is  a function  of  the  length  of  the  conflict  in- 
terval. Second,  consider  an  interval  known  to  contain  at 
least  one  message,  as  in  the  case  when  the  first  part  of  a 
conflict  interval  is  found  to  contain  exactly  one  message. 
Humble t has  shown  that  if  the  length  of  such  an  interval 
exceeds  a certain  threshold,  then  it  is  tetter  to  use  some 
fraction  of  that  interval,  rather  than  the  whole,  as  the 
new  transmission  interval.  The  values  of  these  optimal 
divisions  for  varying  interval  lengths  we re  solved  for  by 
computer  and  the  algorithm  using  these  values  was  found  to 
have  capacity  .43775,  only  marginally  greater  than  before. 

The  reason  for  this  is  that  for  small  interval  lengths  the 
optimal  divisions  are  very  nearly  the  same  as  in  Gallager’s 
algorithm.  In  addition,  it  was  found  that  following  this 
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procedure  would  never  result  in  an  interval  containing  at 
least  one  message  v:hose  length  exceeds  the  threshold  mentioned 
above.  Hence,  after  the  first  division  of  a conflict  in- 
terval, the  algorithms  are  almost  identical. 

1.4  The  General  First-Come  First-Served  Algorithm 

In  this  section  we  describe  the  general  first-come  first- 
served  algorithm.  We  then  examine  a specific  form  of  it 
which  is  an  obvious  extension  cf  Gallager's  and  Kumblet's 
algorithms  and  which  is  amenable  to  analysis. 

For  an  algorithm  to  be  FCF3,  it  must  satisfy  one  of 
two  properties.  It  must  never  allow  a message  to  be  trans- 
mitted when  other  messages  with  earlier  generation  times  must 
wait.  Or,  if  it  does,  the  probability  cf  successful  trans- 
mission must  be  zero. 

Suppose  xhat  we  are  using  a FCFS  algorithm  to  resolve 
conflicts  and  that  all  messages  with  generation  times  prior 
to  some  t have  been  successfully  transmitted.  The  algorithm 
chooses  some  subset  of  the  queue  to  transmit  in  the  next 
slot.  We  call  this  the  transmission  set.  Then  the  most 
general  form  of  a transmission  set  which  satisfies  the  first 
condition  will  clearly  be  the  set  cf  all  messages  generated 
in  an  interval  of  the  form  [t.t-t],  where  x is  some  positive 
real  number. 

Suppose  we  have  a subset  of  the  queue,  3, , which  is 
known  to  contain  at  least  one  message,  tut  it  is  not  known 
whether  it  contains  more.  Consider  the  transmission  set 
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which  is  the  union  of  S1  and  some  subset  of  the  queue,  S2. 

(An  example  of  such  a transmission  set  will  be  given  later 
in  this  section,  in  fig.  2.2.)  If  the  subset  of  S2  which 
is  disjoint  from  is  not  empty,  then  a conflict  occurs 
and  the  second  condition  holds.  If  the  subset  of  S2  which 
is  disjoint  from  S„  is  empty,  then  must  be  the  set  of 
messages  generated  in  an  interval  of  the  form  [t.t-rt)  for 
the  first  condition  to  hold.  Since  we  assume  that  the 
algorithm  never  chooses  a transmission  set  that  is  known  to 
contain  a conflict,  we  cannot  kno w in  advance  if  the 
subset  of  S2  disjoint  from  S1  is  non-empty.  Hence,  S„  must 
be  the  set  of  messages  generated  in  an  interval  of  the 
form  ( t , t-f 1 1 to  insure  that  one  of  the  two  conditions  hold. 

The  general  first-come  first-served  algorithm  must  use 
transmission  sets  which  are  one  of  the  two  types  described 
above.  In  this  thesis  we  restrict  cur  attention  to  the  case 
where  transmission  sets  consist  only  of  the  arrivals  in 
intervals  of  the  form  [t,t+t].  This  is  acne  for  ease  of 
analysis  and  because  it  seems  likely  that  there  is  no  ad- 
vantage in  using  the  more  general  sets.  ;le  call  this  the 
single  interval  algorithm  (SIA). 

Let  us  consider  the  algorithm  in  more  detail.  Suppose 
we  begin  with  an  interval  of  length  s known  to  contain  at  leas' 


one  message, 


’ansnission  interval  will  ; 


a.  For  some  values  of  s we  expect  the  optimal  a to  be  less 


than  or  equal  to  s (fig.  2.1a),  but  in  some  cases  a is  greater 
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than  s (2.1b).  Since  the  case  a<_ 5 has  already  been  discussed, 
we  no w consider  a>s,  say  a=s+t.  If  there  is  exa'"+'ly  one 
message  in  the  interval,  we  are  ready  to  start  a new  epoch. 

If  there  is  a conflict,  and  if  v/e  wish  the  algorithm  to  be 
FCFS , we  have  two  choices.  Our  new  transmission  region  can 
be  either  the  leftmost  part  of  the  interval  of  length  s+t, 
say  of  length  3,  or  it  can  include  the  entire  interval  of 
length  s and  ether  parts  of  the  conflict  and  v/aiting  inter- 
vals (2.2),  as  described  above.  Unfortunately,  in  the  latter 


case,  if  the  transmission  region  contains  a conflict,  the 
definition  of  the  subsequent  procedure  becomes  too  complex 
and  its  analysis  is  difficult.  Hence  we  restrict  our  new 
transmission  region  to  be  an  interval  of  length  2<si-t, 
with  either  2<_s  (2.3a)  or  2>s  (2.3b). 

Let  us  consider  the  case  3<s  first.  When  v/e  transmit 
the  messages  generated  in  the  interval  of  length  £,  there 
are  three  possible  outcomes:  no  message  (2.4a),  one  mes- 
sage (2.4b),  and  a conflict  (2.4c).  If  there  is  no  message 


:hen  we  have  left  an  interval  of  lemm- 


ata mm- 


least  one  message  and  a combined  interval  of  1 eng 
containing  a conflict.  We  process  this  interval 
fashion  that  we  did  the  original  interval,  except 
different  s is  chosen. 

In  the  case  where  one  message  is  found,  we  a 
with  an  interval  of  length  s-rt- 3 which  contains  a 
one  message,  with  the  statistics  derived  below. 


th  s+t- 
in  the 
that  a 


— tr  _ 


t leas' 


ie 
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Throughout  the  rest  of  the  thesis  we  use  a shorthand  notation, 
where  "k  in  s"  is  taken  to  mean  "there  are  k messages  in  an 
interval  of  length  s."  Because  the  generation  statistics  are 
Poisson,  the  numbers  of  messages  in  disjoint  intervals  are 
independent.  Therefore: 

Pr(k  in  s+t-Sji  in  g,  >1  in  s , >2  in  set) 

= Pr(k  in  s+t-6,  1 in  2 , >1  in  s , ^2  in  s-t) 

Pr(l  in  2,  _>1  in  s,  _>2  in  s-rt) 

= Pr ( 1 in  3 )Pr(k  in  s-rt- 3,  >0  in  s-2,  _>1  in  s-t-S). 

Pr(l  in  2)Pr(  0 in  s-2,  in  s+t-2) 

= Pr(k  in  s+t-3,  >1  in  s-rt- 3) 

Pr(  1 in  s-rt- 3) 

= Pr(k  in  s+t-6|_>l  in  s-rt- 3). 

This  result  is  crucial  to  the  implementation  and  analysis  of 
the  algorithm,  since  it  allows  us  to  consider  the  processing 


of 

intervals : one  ■ 

which  contains 

at  least 

ne 

which  contains  a 

conflict,  and 

has  at 

in 

its  first  part. 

After  we  have 

found 

^ f 

length  s-t- 2 con 

tains  at  least 

one  mes- 

sage,  we  process  it  in  the  appropriate  manner,  either  dividin 
it  or  augmenting  it. 

Finally,  if  there  is  a conflict  in  the  interval  of 
length  3,  we  treat  it  as  a conflict  interval  and  return  the 
remainder  to  the  waiting  interval. 

In  the  case  where  3>s,  there  are  two  possible  outcomes: 
one  successful  transmission  (2.5a)  or  a conflict  (2.5b). 


22 


If  we  find  one  message  in  the  interval,  then  we  are  left 
with  an  interval  of  length  s-rt-5  which  contains  at  least 
one  message  and  we  treat  it  as  described  earlier.  When  there 
is  a conflict,  we  have  an  interval  of  length  s containing 
at  least  one  message  and  a combined  interval  of  length  g 
with  a conflict.  We  return  the  remaining  interval  of 
length  s-!-t-2  to  the  waiting  interval,  and  divide  the  interval 
of  length  2 as  described  before. 

1.5  Thesis  Outline 

The  thesis  is  organized  into  three  chapters  and  an 
appendix.  Chapter  1 is  this  introduction.  The  analysis 
is  carried  out  in  Chapters  2 and  3 and  the  computer  pro- 
grams used  in  the  analysis  are  given  in  the  appendix.  Here 
we  consider  the  objectives  and  results  of  these  chapters. 

In  Chapter  2,  we  model  the  single  interval  contention 
resolution  algorithm  as  a Markov  process  with  a continuous 
state  space.  We  define  functions  representing  the  expected 
times  required  to  transmit  all  of  the  messages  generated  in 


a particular  interval  of  time. 


functions  an 


required,  corresponding  to  the  three  classes  of  states  of 
the  Markov  system.  ’We  define  cost  functions  for  the  process 
in  terms  of  the  drift,  the  difference  between  the  lag  at 
the  time  when  the  system  next  enters  3n,  a renewal  state, 
and  the-  lag  at  the  present  time,  he  see  that  the  equations 
for  the  expected  cost  per  epoch  are  identical  tc  those 
developed  for  minimizing  the  time  required  tc  process  an 
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equations  for 


interval.  A technique  for  solving  these 
Humblet's  algorithm  is  described,  and  it  is  explained  why 
this  method  is  unsuitable  for  analysis  of  the  single  interval 
algorithm. 

In  the  third  chapter,  we  describe  some  of  the  basic 
results  of  Karkovian  decision  theory:  the  value  iteration 
algorithm  and  the  Odoni  bound.  We  generalize  these  results 
for  continuous  state  spaces.  We  describe  how  the  value 
functions  may  be  approximated  by  calculating  the  values  for 
a discrete  subset  of  the  state  space.  As  a result  of  com- 
puter analysis  of  these  functions,  we  see  that  for  all 
practical  purposes,  Humblet's  algorithm  is  the  best  steady 
state  single  interval  algorithm  in  terms  of  capacity.  Errors 
in  the  analysis  due  to  the  discretization  of  the  state 
space  are  discussed.  The  insensitivity  of  the  results  to 
the  number  of  points  used  to  approximate  the  state  space, 
leads  us  to  conclude  that  such  errors  are  negligible. 
Suggestions  for  further  research  follow. 

In  the  appendix,  some  notes  on  the  computer  programs 
are  given,  as  well  as  the  program  listings. 
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II.  Analysis  of  the  Algorithm  "by  Expected  Drift 


In  this  chapter  we  model  the  algorithm  as  a problem  in 
Markovian  decision  theory  with  a cost  structure  based  on 
the  difference  in  the  lag  before  and  after  state  transi- 
tions. We  describe  how  this  method  is  used  to  analyze 
Humblet's  algorithm,  and  why  it  is  unsuitable  for  the  analy- 
sis of  the  single  interval  algorithm. 

2.1  The  Algorithm  as  a Markov  Process 


In  this  section  we  model  the  single  interval  algorithm 
as  a Markov  process.  The  state  of  the  system  at  any  time  is 
determined  by  the  length  and  statistics  of  the  previous  trans- 
• mission  interval.  In  section  1.4  we  saw  that  there  are 
three  types  of  intervals  we  will  need  to  consider,  corres- 
ponding to  three  classes  of  states  for  the  system.  The 
system  is  in  state  SQ  if  the  last  transmission  ended  an 
epoch.  The  system  is  in  state  S..(s)  if,  as  a result  of 
the  last  transmission,  we  know  only  that  an  interval  of 
length  s contains  at  least  one  message.  The  system  is  in 
state  SgCs.t)  if  we  know  that  an  interval  of  length  s con- 
tains at  least  one  message,  and  a longer  interval  of  length 
s-t-t  contains  a conflict.  Note  that  the  moments  at  which 
the  system  returns  tc  state  Sn  are  renewal  taints.  When 
the  system  is  in  state  S„,  the  algorithm  chooses  an  inter- 
val  of  length  T as  the  next  transmission  interval.  In 
state  S^(s)  a transmission  interval  of  length  a(s)  is 
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chosen.  In  state  S2(s,t)  we  choose  an  interval  of  length 
6(s,t).  Figure  3 shows  a state  transition  diagram  for  the 
algorithm.  The  number  by  an  arrow  connecting  two  states 
indicates  the  number  of  messages  that  must  be  found  in  the 
transmission  interval  for  that  state  transition  to  occur. 

While  this  Markov  process  has  a continuous  state  space, 
and  therefore  an  uncountably  infinite  number  of  states,  it  is 
v/orth  noting  that  all  but  a countably  infinite  number  of  these 
states  are  transient.  If  the  system  is  in  some  state  other 
than  SQ  at  time  t=0,  then  because  we  assume  the  algorithm  is 
stable,  the  system  must  eventually  enter  3q  . From  SQ,  under 
a stationary  policy,  there  are  only  a countably  infinite 
number  of  states  the  system  may  enter.  This  can  be  seen  by 
observing  that  state  transitions  occur  at  discrete  times,  and 
that  from  a given  state  there  are  never  more  than  a finite 
number  of  transitions  the  system  may  make.  Hence,  all  states 
that  cannot  be  reached  from  SQ  are  transient. 

Having  modeled  our  algorithm  as  a Markov  process,  we 
would  like  to  use  this  model  to  determine  the  values  of 
a(s,A)  and  B(s,t,X)  that  will  maximize  throughput.  (Here 
we  write  x,a  and  B as  functions  of  X to  emphasize  their 
dependence  on  X . ) One  v/ay  of  doing  this  would  be  to  assign 
a cost  or  reward  structure  to  the  state  transitions  and  apply 
the  methods  of  Markovian  decision  theory  to  determine  the 
optimal  controls  r,a  and  3.  In  this  chapter  we  will  consider 
a cost  structure  v/here  we  seek  to  minimize  the  difference 
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in  the  lag  between  successive  epochs.  In  Chapter  3,  we  con- 
sider a reward  structure. 

2.2  Expected  Times  to  Resolve  Intervals 

In  this  section  we  develop  a set  of  equations  whose 
solution  gives  the  optimal  values  of  t(X),  a(s,X)  and  8(s,t,X), 
and  the  capacity,  X*.  Our  approach  is  to  minimize  the  expect- 
ed time  required  to  resolve  conflicts  while  maintaining  finite 
delay.  To  aid  in  the  discussion  we  introduce  the  following 
terminology.  We  say  that  an  interval  has  been  processed 
when  all  the  messages  generated  in  that  interval  have  been 
successfully  transmitted.  We  say  that  an  interval  has  been 
resolved  when  the  system  next  enters  the  state  Sq. 

We  begin  by  defining  three  functions.  Let  MQ(x)  be  the 
expected  time  to  process  an  interval  of  length  x/X  containing 
messages  generated  by  a Poisson  process.  Let  M^(x,y)  be  the 
expected  time  to  process  an  interval  of  length  (x+y)/x , where 
there  is  at  least  one  message  in  the  segment  of  length  x/X 
and  the  arrivals  in  the  segment  of  length  y/X  are  Poisson. 

Let  W^x.y.z)  be  the  expected  time  to  process  an  interval 
of  length  (x+y+z)/X,  where  there  is  at  least  one  message  in 
the  segment  of  length  x/X  , at  least  two  in  the  segment  oi” 
length  (x+y)/X , and  the  arrivals  in  the  segment  of  length 
z/X  are  Poisson.  The  variables  x,y  and  z may  be  thought  of 
as  the  lengths  of  the  intervals  in  a normalized  time  frame 
where  the  Poisson  arrival  rate  is  1 message  per  time  unit. 

It  is  important  to  note  that  though  the  arguments  of  these 
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functions  are  given  in  normalized  time,  the  values  of  the 
functions  are  in  real  time.  If  T,  A and  B are  the  normal- 
ized lengths  of  the  transmission  intervals  used  when  the 
system  is  in  the  states  Sq,  and  S2  respectively,  and  if 
T is  fixed  and  A and  B are  chosen  optimally  for  that  T, 
then  from  these  definitions  and  from  the  discussion  of 
section  1.4  we  have: 


(2.1a)  MqT(x)  = 1 + Pr  (>2  in  (x,0,0)  + Pr(0  or  1 in  T)MQ‘(0) 


X<T 


1 + Pr(_>2  in  T)M2T(T,0,x-T)+Pr  (0  or  1 in  T)M0T(x-T)  x>T 


(2.1b)  M^T(x,y)=min /l+Pr  (0  in  A|d  in  x)M^T(x-A,y)+ 


Pr(l  in  Apl  in  x)M  (x+y-A)+ 

Pr  ( j>2  in  A j _>1  in  xJJl^  (A,0,x+y-A)  A^x 
1 T 

1+Pr(l  in  A (_>1  in  x)M^  (x+y-A)+ 

\ > T 

\ Pr (_>2  in  A Pi  in  x)M_  (x,A-x,x+y-A)  A>x 


(2.1c)  M_T(x,y,z)=min/ 1+Pr  (0  in  E '_>1  in'  x,>2  in  x+y)H  ‘(x-B,y,z)  + 

B I T 

Pr(l  in  B j d in  x,^2  in  x+y ) (x+y-B,z)+ 

1 T 

Pr(_>2  in  Bpl  in  x,_>2  in  x+y)M2  (3,0,x+y+z-3)  B<x 
1+Pr(l  in  B|_>1  in  x,p2  in  x+y)M2T (x+y-B, z) + 

Pr(p2  in  b|_>1  in  x,p2  in  x+y) M2T (x, A-x,x+y+z-A) 

x<3<  x+y 

Nov/  we  define  three  new  functions,  F„ , F.  and  in  terms 

U i c. 

of  Mq,  Mx  and  Mg.  These  functions  are  also  dependent  on  T, 
but  we  do  not  write  the  T for  simplicity.  Let: 
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(2.2a)  MqT(x)  =■  F (x) 

(2.2b)  H*(x,y)  * F1  (x)  + FQ  (y) 

(2.2c)  M2T(x,y,z)  = F2<x,y)  + FQ(z) 

Combining  eqs.  2.1  and  2.2  we  get: 


(2.3a)  FQ(x)=  1+FQ(0)+Pr (>2  in  x)F_,(x,0)  x<T 

1+Fq (x-T) +?r (>2  in  T)F  (T,0)  x>T 


(2.3b)  F^ (x) +Fq (y) =min  ^ 1+Pr(0  in  Aj_>l  in  x) (F^  (x-A) +F Q (y) ) + 

Pr(l  in  a[>1  in  x)F„(x+y-A)+ 


J Pr(>2  in  a|>1  in  x) (F, (A,0)+FQ(x+y-A) ) A<x 

1+Pr(l  in  a|>1  in  x) F^ (x+y-A) + 

s.  Pr(>2  in  A*>1  in  x)  (F  (x,A-x)+F„  (x+v-A) ) A>x 
x — — 2 O' 


(2.3c)  F2 (x,y) +Fq (z) = 


min/’l+Pr(0  in  b|_>1  in  x,>2  in  x+y)  (~2  (x-5,y)  +F,  (z) ) + 

Pr(l  in  B j >1  in  x,>2  in  x+v) (F,  (x+v-3) +?, fz) ) + 

~~  — 1 “ • 0 

J Pr(>2  in  B : >1  in  x,>2  in  x+y) (F_ (B, C) +F- (x+v+z-S)  B<x 
1+Pr(l  in  Bj_>l  in  x,>2  in  x+y)  (F  (x+y-3)  +Fq  (z) ) + 

^ Pr(^2  in  b|>1  in  x,_>2  in  x+y)  (F2  (x , B-x)  +F^  (x+y+z-B) ) 

x<B<x+y 


Since,  for  x T,  Fq(x)-Fq(x-T)=1  - Pr(>2  ir.  T)F.,(T,0)  = 
a constant,  F,(x)  is  asymptotically  linear  in  x.  Hence, 
for  large  x,  we  may  write  ?0(x)=kx/A  + b.  Let  k*=k/X  . 

Then  the  system  will  be  stable  so  long  as  k*  is  less  than  l/X . 
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We  may  use  this  expression  for  FQ  to  rewrite  eq.  (2.3a): 

k*x  = 1 + k*(x-T)  + Pr(^2  in  T)F2(T,0)  or 

(2.4)  k*  = 1 + Pr (22  in  T)F^(T.O)  l/X 

T 

We  may  also  rewrite  eq.  (2.3b): 


F (x) +k*y+b=min 
A 


1+b+Pr  (0  in  A |_>1  in  x)  (F  (x-A)  +k*y)  + 

Pr(l  in  a|>1  in  x) (k*y+k* (x-A) ) + 

Pr(>2  in  A;>1  in  x) (?2 (A, 0) +k*y+k* (x-A) ) A<x 
1+b+Pr  ( 1 in  a|>1  in  x) (k*y+k* (x-A) ) + 

Pr(>2  in  a|>1  in  x) (F  (x,A-x) +k*y+k* (x-A) ) A>x 


or 


(2.5a)  F^(x)=min  C 1+Pr(0  in  a|>1  in  x)F^  (x-A)+Pr  (>2  in  a!>1  in  x)F2(A,0)+ 
J (1-Pr(0  in  A j >1  in  x))k*(x-A)  A<x 


l 


1+k*  (x-A)  +Pr  (>2  in  A [_>1  in  xjF^tx.A-x)  A>x 


Similarly,  v/e  may  rev/rite  eq.  (2.3c) 


(2.5b)  F2(x,y)=nin 


r 


l+?r(0  in  Bj^l  in  x,^2  in  x+y)F  (x-S,y)+ 
Pr(l  in  b|>1  in  x,^2  in  st+y)F  (x+y-3)  + 


l Pr(^2  in  b|^1  in  x,>2  in  x-*-y)  (F  (3,  0)  +k*  (x+y-B) ) B<x 


^1+Pr(l  in  B j >1  in  x,^2  in  x+y)F^  (x+y-B)  + 


Pr(>2  in  B !_>1  in  x,>2  in  x+y) (F  (x,S-x)+k* (x+y-B) ) 

x<B<  x+y 
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Hence  v/e  wish  -to  find  the  smallest  value  of  k*  such  that 
v/hen  A(x)  and  B(x,y)  are  chosen  to  minimize  F^  (x)  and  F2(x,y) 
there  exists  some  value  of  T for  which  eq.  (2.^)  is  satis- 
fied. Then  the  capacity  will  he  X*=l/k* . 

2.3  Minimizing  Expected  Drifts 

In  addition  to  their  definition  in  terms  of  expected 
times  to  resolve  intervals,  the  functions  Fq(x),  F^(x)  and 
F2(x,y)  may  also  be  interpreted  in  terms  of  the  expected 
drift,  which  we  define  below. 

We  define  a random  variable  U(S),  called  the  drift.  If 

the  system  is  in  state  S at  time  t,  then  U(S)  is  the  differ- 
ence between  the  lag  at  the  time  v/hen  the  system  next  enters 

the  state  SQ  and  the  lag  at  time  t.  Let  Uq(Sq) , U (x)  = 

U(S«(x/X  ))  and  U2(x,y)=U(32(x/\  ,y/X  ) ) . UQ  is  the  drift  in 
lag  between  one  epoch  and  the  preceding  epoch.  If  the 
system  is  to  be  stable,  then  it  is  clear  that  E(Uq)<0, 
otherwise  the  delay  will  become  infinite  as  the  number  of 
epochs  goes  to  infinity.  The  capacity  of  the  algorithm 
will  be  the  largest  value  of  \ for  which  a policy  exists 
that  has  E(TJ q)  less  than  zero. 

Next  v/e  show  that  F^  (x)=E(U^  (x) ) + x/>.  and  F2(x,y)  = 
S(lr2(x,y))  t (x-!-y)/\  . Therefore,  minimizing  F^  and  F?  for 
a fixed  \ is  eauivalent  to  minimizing  the  drift.  To  see 


this,  consider  Figure  4.  A' 


►.1C  o w** 


**  s>  rs  • -rs  ~ c,1- 


5 with  a transmission  interval  of  length  1 and  lag  d.  A' 
time  t'  the  system  has  returned  to  with  lag  d' . The 


length  of  the  processed  interval  is  1' . 


Then  we  have 


U(S)=  d'-d  = (t'-t)-l’ 

and 

E(U(S ) ) + 1 = E(t'-t)  + E(l-l’) 

= E(time  to  return  to  SQ)  + 

E(tine  returned  to  the  waiting  interval)  - 
E(time  taken  from  the  waiting  interval)  . 
Examining  ea.  (2.5)  we  see  that  F.  and  F2  may  also  be  de- 
fined as  the  expecte^  time  to  resolve  an  interval,  plus  the 
time  returned  to  the  waiting  interval  and  minus  the  time 
taken  from  the  waiting  interval. 

With  this  interpretation  for  F2(x,y),  it  is  easy  to 
see  that: 

E(U0)  = 1 - Pr(>2  in  T)F2(T,0)  - I/;  . 

Hence  the  condition  that  E('JQ)  be  less  than  zero  is  equiva- 
lent to  the  condition  of  ea.  (2.4). 

2.2  Solving  the  Equations  for  U 2 , F, (x)  and  ?2(x,y) 

In  this  section  we  discuss  methods  fcr  solving  eqs. 
(2.4)  and  (2.5).  In  their  present  form  these  equation 


o dl  bt 


very  difficult  to  work  with.  To  simplify  them,  v/e  define 
two  new  functions: 

g.  (x)=  Pr(>l  in  x)F^(x) 

62(x-H)=  ?-(>'-  in  l2  in  x-y)F2(x,y) 

Substituting  these  equations  into  ea.  (2.4),  the  stability 
condition  becomes: 

(2.6)  1 + g2(T,0)  - T/A  = 0 . 
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From  ea.  (2.5)  we  get: 


(2 


.7a)  (1-e  X)  + e Ag  (x-A)  + g2(A,0)  + (1-e  A)  (x-A) 

A S -x , . , . . . X* 


A<x 


(1-e  X)(l-(A-x))  + g? (x,A-x)  A>x 


X* 


(2.7b)  g2(x,y)=min 
B 


-X 


-v. 


-B 


(1-e  (1+xe  -))  + e g2(x-B,y)  + g (3,0)  + 


Be  Bg  (x+y-B)  + (1-(1+B)e  3)  (x+y-3) 


B<X 


X* 


(1-e  X (1+xe  7) ) + xe  Bg1 (x+y-B)  + g2(x,B-x)  + 


(1-e  X-xe  B) (x+y-B) 


x<B<x+v 


X* 


First  we  consider  the  problem  cf  solving  these  equations 
for  the  case  where  we  restrict  A <_x.  (This  is  the  method 
used  by  Humble t to  analyze  his  algorithm. ) Then  the 


second  argument  of  will  alwavs  be  0 and  we  also  have 

°2 


B<x.  Thus  we  see  that  g1 (x)  ana  g-(x,0)  are  defined  in 
terms  of  g1  and  g-  at  smaller  arguments.  In  addition,  i" 


is  easily  verified  that  g„(C)=0  and  g_(C,C)=T.  Thus  we 

i ^ 

may  construct  the  functions  g^  and  ^7  starting  with  seme 
very  small  x and  solving  simultaneously  for  g. (x)  and 

j. 

g2(x,0),  linearly  interpolating  between  g1 (x)  and  g, (0), 


and  £- (x , 0 ) and  £-(0,0),  where  needed.  Vj e then  crcceed 


by  incrementing  x and  solving  an  each  seep  by  interpolating 
between  the  previously  calculated  values.  This  v/ill  lead 
us  into  trouble,  however,  since  it  can  be  shewn  that  for 


small  values  of  x,  g2(x,0)  is  quadratic  in  x,  not  linear. 
To  circumvent  this  difficulty,  we  define  a new  function, 
g2  (x,y)=g2(x,y)/x.  The  changes  to  eq.  (2.7)  are  obvious 
and  we  do  not  present  them  here. 

The  following  argument  shows  that  g1 (x)  is  linear  in 
small  xand  that  g2(x,0)  is  quadratic  in  small  x.  As  x 
goes  to  0, 

Pr(k  in  x |_>1  in  x)  =j  1 if  k=l 

\o  if  k/l 

Therefore,  the  expected  length  of  time  required  to  process 
this  interval  is  just  one  slot;  that  is,  F^(0)=1.  So  for 
small  x,  g1(x)=  l-e-x=x. 

Also,  as  x goes  to  0, 

Pr(k  in  x|_>2  in  x)  = fl  if  k=2 

lo  if  k/2 

For  small  x,  the  optimal  B is  approximately  x/'2.  Then, 
Pr(0  in  x/2 1 _>2  in  x)=  Pr(>_2  in  x/2|_>2  in  x)=  f 

and 

Pr(i  in  x/2’>2  in  x)  = % 

From  eq.  (2.5b)  we  get 

F2(0,0)  = 1 + iF2(0,0)  + tF.(O) 
and  therefore 


f2(o,o)=  3 . 

So  for  small  x,  gp (x, 0 )= 3 ( 1- ( 1-x) e~A)* 3( 1- (1 
(3/2 )x2. 


Humblet  has  written  a computer  program  to  construct  the 
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funtions  g^(x)  and  g^(x,0)  by  this  technique.  After  the 
functions  are  calculated,  the  drift  is  found  from  gp.  Dif- 
ferent values  of  X are  used  to  determine  the  highest  value 
that  gives  a non-positive  drift,  E(UQ).  The  optimal  initial 
interval  length,  T,  is  found  to  be  approximately  1.27  and 
the  throughput  is  approximately  .438. 

This  method  has  several  problems.  One  problem  is  that 
the  program  cannot  solve  for  X*  direcxly.  If  must  be  run 
for  many  different  values  of  X to  find  xhe  greaxesx  one. 
Also,  solving  for  more  than  two  decimal  places  of  accuracy 
in  the  optimal  initial  interval  length  requires  an  excessive 
amount  of  computation,  especially  in  view  of  that  fact  that 
the  program  will  be  run  many  times  to  find  the  capacity. 
Finally,  because  the  program  relies  on  finding  \*  such 
that  the  drift  is  zero,  the  result  is  very  sensitive  to 


small  errors  in  g.  and  av. 

=1  ~2 

Attempts  were  made  to  solve  for  the  capacity  by  this 
method  when  A is  not  restricted  to  be  less  than  or  equal  to 
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III.  Analysis  of  the  Algorithms  by  Markovian  Decision  Theory 

with  a Reward  Structure 


In  this  chapter  we  analyze  the  algorithms  by  use  of  a 
reward  structure,  where  a reward  of  one  is  given  for  every 
successful  transmission.  We  want  to  maximize  the  expected 
reward  per  state  transition,  which  is  called  the  gain. 
Clearly  the  gain  is  equal  to  the  throughput  and  the  maximum 
gain  is  the  capacity.  Before  proceeding  with  this  analysis, 
we  review  some  basic  results  of  Markovian  decision  theory 
which  were  derived  for  Markov  processes  with  finite  state 
spaces,  and  show  how  they  can  be  generalized  to  the  continu- 
ous state  space  model. 

3.1  Markovian  Decision  Theory  and  Continuous  State  Spaces 
In  this  section  we  introduce  two  basic  methods  cf 
Markovian  decision  theory,  the  value  iteration  algorithm 
and  the  Odoni  bound,  and  generalize  then  for  continuous 
state  spaces. 


3.1.1  The  Value  Iteration  Almorithn 

The  object  of  Markovian  decision  theory  is  to  find 
optimal  policies  for  controlling  a Markov  process  in  order 
to  maximize  the  gain.  Cr.e  method  of  determining  such  a 
policy  is  through  use  of  the  val.,e  iteration  equations  (?). 
The  following  model  is  used.  We  have  a finite  state  space, 
with  states  i=l , . . . ,N,  and  for  each  state  i,  a finite 


choice  of  controls  k=l , 


, n^.  We  have  a set  of  state 
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transition  probabilities , p-  . , which  are  dependent  on  the 

J 

control  k used,  and  a set  of  state  transition  rewards,  r.  .*  , 

J 

which  may  or  may  not  be  dependent  on  the  control.  The 
following  quantities  are  defined: 


(3.1) 


k_  y k k 

1 j=1  pij  ij 


<3-2>  Vik(n+1)(n+l)=  + E pi3 

J ^ 


k(n+l)  k.(n)/  ) 

3 j ' 


Equation  (3-1)  gives  the  expected  reward  on  the  next 
transition  if  the  system  is  in  state  i and  control  k is 
applied.  Equation  (3-2)  defines  the  value  functions,  where 
k(n+l ) ^ -t0paq  expected  reward  ever  the  next  n+1 

transitions,  if  the  system  begins  in  state  i and  the  control 
k(n)  is  applied  when  there  n transition  remaining.  The 
values  of  v^(0)  are  assumed  to  be  given  and  are  called 

0 v»r*4  ^ v“fc3.  ^ P V/  3.  C*  3 

If  the  values  of  k-  are  chosen  at  each  r.  such  that 


(3-3) 


vi(n+l)  = 


max  q; 


N k 

■ Z pii 
j=l  1J 


v.(r.) 


then  the  sequence  of  vectors  k(n)  represents  an  optimal 
time- dependent  policy.  This  is  the  optimality  principle, 


If  the  values  of  k are  held  constant  for  ail  n,  we  s 


^ * * ci  1 


the  policy  is  stationary.  Schweitzer  (10)  has  she wn  that, 
for  any  set  of  initial  values  v^(0),  if  all  stationary 
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policies  result  in  single-chain  aperiodic  Markov  processes, 
then  as  n-+«,  v^(n)->-ng+v^ , where  g is  the  gain  and  the  v^ 
satisfy  s 

N 

(3**0  v.=  max  q.k  - g + I p.  .k  v. 

1 k j=l  1,3  J 

In  addition,  any  vector  k which  satisfies  this  equation 
defines  a policy  with  the  maximum  gain. 

Schweitzer  also  introduces  a notation  for  considering 
a continuous  state  space  with  finite  volume.  Careful 
examination  of  his  proof  of  the  above  indicates  that  a 
similar  theorem  for  continuous  spaces  could  be  shown  in  an 
analogous  manner  using  the  general  notation,  and  adding 
some  simple  assumptions  about  continuity.  T;/e  make  the 
following  conjecture. 

':ie  consider  a system  with  a state  space  3,  which  is 
the  union  of  a finite  number  of  sets  with  finite  volume. 

That  is. 

In  n. 

3=  S.  where  S.eR  and  dx  <*  . 
i=i  - ~i 


The  state  of  the  system  is  represented  by  a vector  x;s5., 


l<i<_N.  vMote  that  if  the  sets  3^  are  disjoint,  the  sub- 
script of  x may  be  omitted.)  For  each  state  there  is  a 
set  cf  controls  K(x^).  From  each  state  x;c3.,  for  each  con- 
trol k:  F. v>:.  ) , for  each  j=l,.  . . there  exists  at  most 

one  point  y.eS.  which  the  system  may  enter.  The  control  k 

J *j 

• • 1c 

which  is  chosen  determines  a set  of  probabilities  . (x), 


serin-: 


set  cf 


trol  :U 


one  pci 
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where  P.  -K(x)  is  the  probability  that  the  system  will  go 

^ J 

from  state  x^S.  to  state  yeS.  if  the  control  k is  applied. 

J 

The  reward  functions,  r.  .k(x)  are  defined  in  the  obvious 

• J 

manner.  With  this  notation,  we  rewrite  eqs.  (3.1),  (3-3) 
and  ( 3 . 4 ) . 


(3.5) 

(3.6) 

v. (x, n-1 ) = sup  q. 

1 k 1 

(3.7) 

V. (x)  = sup  q . ‘ (x)  rn 

-4  U 

k, 


N 


J=1 


N 


k 


y 


The  sup  in  eqs.  (3.6)  and  (3.~)  can  be  replaced  by  max 
if  we  assume  either  that  the  control  sets  are  finite,  or  if 
they  are  compact  and  the  right  hand  sides  cf  the  equations 
are  continuous  in  k for  ail  points  ir.  the  state  space. 

We  conjecture  that,  for  any  initial  set  of  values 
v. (x,0),  if  all  stationary  policies  result  in  single  chain 
aperiodic  Markov  processes,  then  as  it-cc , v.  (x,n)~  ng-v.  (x), 
where  g is  the  gain  and  the  v^  satisfy  eq.  (3.?). 

'While  we  do  not  attempt  to  prove  this  conjecture  here, 
it  will  be  seen  later  that  when  we  oalculate  approximations 
to  the  v.  ( x , r. ) for  our  crotlen,  tn.  ee  value  functions  do 
indeed  converge. 
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3.1.2  The  Odor.i  Bound 

Odoni  (11)  has  provided  upper  and  lower  bounds  for  the 
gain  that  can  be  calculated  at  each  iteration  of  the  value 
iteration  algorithm.  We  can  use  these  bounds  to  give  an 
estimate  of  the  gain  and  upper  bound  the  error  of  our 
estimate.  These  bounds  generalize  easily  to  the  continuous 
state  space.  For  the  continuous  state  space  model  described 
in  the  previous  section,  v/e  give  the  following  theorem. 

Theorem.  Let  (x)  and  v^(x,n)  be  defined  as  in  eqs. 

(3.5)  and  (3-5).  Let  v. (x,0)  and  r-  -(x)  be  bounded  for  all 
l<i  <N,  x^eS. . Then, 

i)  L' ' (n)=  sup  (v.  (x,n-rl  )-v.  (x,n) ) is  monotone  decreasing  in  n. 

i,x 

ii)  L' (n)=  inf  ( v^ (x, n-1 ) -v^ (x, n) ) is  monotone  increasing  in  n. 

i , x 

iii ) L'(n)<lim  inf  v.(x.n')  < lim  sup  v.  (x,n')  <L' ' (n) 

n'-*00  V — 

n 11  for  all  n. 

Proof.  The  proof  of  i)  ana  ii)  is  completely  analogous  to 

the  proof  given  by  Cdoni,  and  we  do  not  repeat  it  here.  To 

prove  iii)  observe  that  for  fixed  n,  for  any  n’>n, 

vi(x,n’ )=(vi(x,n' )-vi(x,n'-l) )t  . . . 

+(v. (x,n+l)-v. (x,n) ) -r  v. (x,n) 

-L  J.  X 

< sup(vi(x,r.’ )-v.  (x,n'-l) )-  . . . 

msup(v,  (x,n-l )-v.  (x, r.) )—/.  (x.n) 

= L"(n’-1)+  . . . +L'  ’ (n)-v.  (x,n) 

_<  (n'-n)L’'(n)  + sup  v^(x,n)  1 
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this  chapter  differs  from  that  used  in  Chapter  2,  in  tha 


the  variables  x and  y here  represent  the  normalized  lengths 
of  intervals,  that  is,  the  units  in  which  x are  measured 
are  chosen  such  that  the  arrival  rate  of  the  messages  is  1. 
The  controls  T,  A and  B are  normalized  lengths  also. 

The  reward,  which  is  independent  of  the  control,  is 
1 for  a successful  transmission  and  0 otherwise . Hence  the 
expected  reward  on  the  next  transition  is  just  the  probabili- 
ty of  finding  one  message  in  the  transmission  interval. 

From  eqs.  (3 -5)  and  (3*-)  we  haves 


(3.8a)  v~  (n+l)=  max  Te“T+(l-KT) e” "v-  (n) 


T>0 


♦ <l-<l+T>e-)vS2(Ti0)<n) 


V _ £ _ y 

(3.8b)  v„  / -,  ( r.ml ) = max/  v-  , ,anjr  «e 

^x;  A>0[  . -x  ' 


(n)-  Ae~H  (Itv,  (n) ) 


.-e 


-A 


m 1-  ( 1mA ) e V-.  /. 

— z2{A,0) 

1-e  X * 


i-e 

(r.)  A<x 


^ xe~A  (1-v,  (n)) 
l-e-x  30 


+ l-e-x-xe_A  v_  , „N(n)  A>x 
^2u*a-x) 

s 1-e 


(3-3c)  v3  . .rUr-“S)-  e ~-s  ~''v-  , _ ,.\(n. 

X'g  ' 0<E<y  , ■ 


^ ' y_  — • *■  ; 

*^2  vA  - ' j f 


^Ble"3-e~(x~y))(l-v,  , ...  . ■ ( n ) ) 


l-e_x-xe”  ^x"r,>  ^ 
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. 


! + 1 - ( 1 -*-3 ) e 


-3 


vS.(B,0)(n)  E<x 


(e-B.e- <*+?>)( i,vf  ,.„._  = )(n)) 

(n) 


l-e_x-xe 


+ l-e”X-xe 


=rsyT  si(-y-B> 

-B 


v 


,.6-x.M-UT-)  2 


3~(x,E-x) 


V 1-e 


x<B<x*ry 


We  use 


initial  values  Vg^Oj-v^  (x)  (0)=0 

for  ail  x>0 , y>  0.  VJith  this  choice  of  terminal  rewards , 

we  may  write  closed  form  expressions  for  v0  (l)  and 

°0 

S2(x,y)(1)  ar'd  not  fdr  ar-y  of  the 
value  functions  when  n>l.  Therefore  we  have  written  a 


vs  (x)(l),  but  not  for  v, 


computer  program  to  solve  for  the  value  functions. 

The  program  can  calculate  the  value  functions  only  for 
a finite  number  of  points.  Hence  we  approximate  the  state 
space  with  a discrete  grid.  Since  we  know  that  at  each 


set  i i | _ 

It  should  be  clear  that  for  the  state  S.(x)  we  will  never 

1. 

v/ant  to  choose  a transmission  interval  longer  than  T'  , and 
we  may  also  limit  the  control  set  for  S1 . Noting  that  the 
control  set  for  S2(x,y)  is  already  bounded,  we  see  that  the 
sets  of  non— trans i ent  states  are  bounded,  sires  the  system 
never  has  to  consider  an  interval  lor.rer  than  T‘  . Hence 
our  discrete  approximation  to  the  state  spa: 
number  of  points. 
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has  a fir.i 


re  is  some  finite 

optimal 

initial  in 

tervai 

we  may  limit  the 

control 

set  for 

to  the 

<«},  where  T'  is 

chosen  s 

uch  T(n)<T' 

for  all  n. 

When  writing  the  computer  program,  it  was  necessary  to 
guess  in  advance  approximately  where  the  bounds  on  the  set 
of  non- transient  states  lie.  Since  v/e  can  only  store  a 
small  number  of  points,  and  since  v/e  would  like  the  grid  to 
be  as  fine  as  possible  for  accuracy,  it  is  important  to 
choose  the  bounds  on  the  grid  well.  The  choice  of  these 
bounds  is  discussed  in  the  Appendix. 

Since  v/e  can  only  calculate  the  functions  at  a finite 
number  of  points,  and  since  this  calculation  will  often 
require  values  of  the  functions  from  the  previous  iter- 
ation at  points  which  are  not  stored  in  the  grid,  it  is 
necessary  to  approximate  the  values  at  these  points.  The 
program  does  this  by  linearly  interpolating  between  the 
values  of  the  two  points  on  the  grid  which  are  closest  to 
the  point  whose  value  v/e  are  estimating.  This  method  intro- 
duces some  error  into  the  calculation  of  the  value  functions 
It  is  difficult  to  analyze  this  error,  but  the  results  of 
the  computer  program  indicate  that  is  quite  small  for  even 
very  coarse  grids,  be  discuss  this  in  more  detail  in  seotic: 


3*5. 


This  method  of  estimating  the  values  of  coints  not  on 


:he  £rid  creates  a problem  when  we  calculate  th 


:tions  at  S.(ix)  or  S , ( lx , y ) , where! 


* c ST7.21-.  — 


l • 

X 


x 0 ^(ix)^  a-"  ' S 2 (1  x , y ; ' ‘ " 

both  require  knowledge  of  v5^(x)(^)  and  v,  y) (n)  for 

values  of  x<Ax.  But  there  are  no  points  on  the  grid  below 
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vs  (yx)(n)  an^  v3  ( v \ (-n ) "to  use  in  the  interpolation. 
Hence,  we  define  vs  (0)(n)=  lim  v (x)(n)  and  vs  (0>y)(n)= 

X X W X w 

“S  VS2(x,y)(n)  for  ^°-  We  let  ^(Oj^’^S^O.y)10^0' 
From  eqs.  (3.8b)  and  (3.8c)  we  get: 

(3-ed)  vS1(0)(n"1)=  e-'-(l+vSoin))  ♦ (l-e-A)vS2(0iA)(n) 

- B -v 

^ »n  4- 1 ^ — mo  v o 4-”tr  . I r-k  ^ ^ 4- 


(3.3e)  v„  rn  (n+1  )=  max  e -e~J  (l+v~  , M(n))  + 

iS2'u,i^  B>0  Tty 

— i - e J 


1-e  y 

Defining  vg  ^(n)  presents  a problem.,  since  the  value 
function  is  discontinuous  at  this  point.  That  is, 


lim  v. 


x-Q  2 


S,(x.O)(n)  * Jl5  %(0,y)<B> 

y— u ^ 


As  x goes  to  0,  the  state  S2(x,0)  represents  the  knowledge 
that  a very  small  interval  contains  two  messages.  As  seen 
in  section  2.4,  it  will  require  an  average  of  3 slots  to 


process  these  messages. 


as  y goes  to  ~ , tne  state 


S 2 ( 0 , y ) represents  the  knowledge  that  two  very  small  inter- 
vals each  contain  one  message,  requiring  only  2 slots  to 
process.  Hence,  the  discontinuity  of  the  value  functions 
arises.  To  overcome  this  problem,  the  program  is  written 

to  interoolate  between  the  values  lim  v-  . ^(m)  and 

x-*0  Z2K‘"  ' 

v-  / . ..  n ( r. ) whenever  a value  for  vr  , - ,(r.  is  needed 
where  0<x<ix.  Similarly,  the  program  interpolates  between 
lim  vs  (0,y)(n)  and  vsp(0,hy)(n)  wheneve-  a value  for 

y 0 2 47 


and  then  the  optimal  stationary  policy.  Listing's  of  the 
programs  and  some  descriptive  notes  are  given  in  the 
Appendix. 

3.3.1  The  Optimal  Time  Dependent  Policy 

The  program  results  show  that  for  a finite  horizon, 


for  the  state  31 (x) , the  optimal  control  A is  greater  than 
x for  x less  than  a threshold  deoer.dent  :n  n , which  v;e  cal 


Xm(n).  The  control  function  A(x,n)  is  discontinuous  as  xT(n). 


For  x less  than  x^(n) , A is  increasing  in  x and  decreasing 
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in  n.  The  threshold  is  also  decreasing  in  n.  This  is 


illustrated  in  Figure  5.  which  shows  plots  of  A(x)  for 
n=3,l  and  5-  It  was  not  possible  to  plot  A(x)  for  n 
greater  than  5.  since  x,p(n)  for  larger  n is  less  than  the 
smallest  positive  value  of  x for  which  v~  , ,(n)  is 

^ A V ^ ) 

calculated.  However,  for  all  successive  n,  the  program 
shows  A(0)>0,  with  A(0)  converging  to  .0,5?. 

We  conjecture  that  for  all  finite  n,  there  exists 
some  positive  x such  that  fcr  x'  less  than  x,  A(x,,n)>x*. 

This  need  not  he  true  as  n goes  to  infinity,  even  though 
A(0)>0  for  all  n,  since  the  control  function  is  discontinuous 
The  discontinuity  of  A(x)  stems  from  the  fact  that  the 
function  v,  , (r) , when  considered  as  a function  cf  A 

o ^ V ** ) 

for  fixed  x and  n,  has  two  local  maxima.  (Tee  Figure  c.) 

a 

For  example,  v_.  , nMA(3)  has  local  maxima  of  1 .7^55  at 
A= . 06  and  1.7b92  at  A=.3,  and  hence  the  optimal  control  is 
greater  than  x.  The  threshold  fcr  n-3  occurs  ax  r~  07, 
where  the  local  maxima  are  hoxh  1.7125  at  A=.07  and  A=. 

For  x~.CS,  the  maxima  are  1.7102  at  A=.C5  and  1.7316  at 
A=.32.  Hence  we  see  a discontinuity  in  the  control  function 
at  x=.07,  out  not  in  the  value  function. 

For  the  states  S.  (x),  there  is  a second  threshold,  x~,, 

such  that  for  x-,<x<x_II  , A(x)=x,  ’cut  for  x>  x, , , A (x)<x. 


For  large  n,  xT,= 


— friv.  v o — 


oth  Humble t*  s algorithm  an i the 


3IA . 


?he  slope  of  A(x)  is  discontinuous  at  this  coin: 


It  was  this  fact  that  originally  led  us  to  suspect  that 


ft 


we  might  want  A(x)>x  for  x<.73.  We  expected  that  letting 
A exceed  x for  x<.78  would  remove  this  discontinuity  in  A(x). 

The  controls  for  the  states  S^Cx.O)  are  also  the  same 
as  for  Humblet's  algorithm.  In  addition,  it  was  found 
that  B(x,y,n)  was  always  less  than  or  equal  to  x.  for  y<. 96. 
3-3-2  The  Optimal  Stationary  Policy 

The  program  written  to  calculate  the  value  functions 
was  designed  to  stop  when  the  difference  between  the  upper 

_ C 

and  lower  bounds  for  the  gain  became  less  than  10  This 
occurred  after  IS  iterations.  Vj e assume  that  the  policy 
at  this  point  is  the  optimal  stationary  policy.  This  policy 
for  the  single  interval  algorithm  is  indistinguishable 


from  the  policy  for  Humblet's  algorithm. 


is  not  clear, 


however,  that  the  algorithms  are  identical.  As  mentioned 
in  the  preceding  section,  for  n>5,  x_(n)  is  tco  small  for 
the  program  to  calculate.  This  threshold  decreases  so 


rapidly,  that  we  conjecture  that  xm(n)-0  as  h-*-00 . Even  if 


this  is  not  true,  the  vaiu- 


xT  to  which  X- ( r. ) 


c c river- es 


is  sc  small,  that  the  probability  that  the  system  will 


ever  enter 


a state  S1 (x) , where  x<x„  is  nearly  zero.  Thus, 


for  all  practical  purposes,  Humblet’s  algorithm  is  the 
optimal  steady  state  single  interval  algorithm. 


As  mentioned  earlier,  using  Humblet's  algorithm, 
the  system  will  never  encounter  a state  S,  (x)  where  7‘ 


Therefore,  A=x  for  all  non- transient  states  S. (x).  This 


fact,  and  a table  of  values  for  B(x)  completely  specify. 
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ncTio.-s  vSi(x)(l)  ar.d 


the  algorithm.  A table  of  3(x)  is  included  on  the  next 
page. 

3.^-  Error  in  the  Value  Functions  due  to  Linear  Approximations 
In  this  section,  we  explain  why  we  believe  that  the 
errors  introduced  into  the  value  functions  by  linear  inter- 
polation are  negligible 

If  the  function  being  approximated  is  very  nearly  linear 
over  the  region  between  the  points  where  it  is  being  inter- 
polated, then  the  error  in  the  interpolation  will  be  small. 
Even  at  the  first  iteration,  the 

Vg  are  very  nearly  linear.  After  the  program 

converges  to  a stationary  policy,  v~.  , %(n)  is  even  closer  to 

^ V X / 

being  linear  and  v^  ^(n)  is  nearly  linear  in  both  x and 

y. 

If  the  function  is  monotonic  between 
the  points  where  it  is  being  interpolated,  the  error  will 
be  no  greater  than  the  difference  between  the  values  of 
the  function  at  those  two  points.  This  difference  may  be 
made  as  small  as  desired  by  choosing  the  grid  fine  enough 
with  respect  to  the  slope  of  the  function.  Since  the  slope 
of  the  functions  is  low,  we  can  use  a relatively  coarse 
grid  and  still  keep  errors  small. 

A powerful  argument  for  neglecting  possible  inter- 
polation errors  is  that  program  results  are  almost  completeiy 
independent  of  the  size  of  the  grid  used.  In  particular, 
a program  was  written  to  analyze  Gallager's  algorithm 
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X 

B 

0.01 

0.005 

0.44 

0.214 

0.8? 

0.410 

0.02 

0.010 

0.4  5 

0.218 

0.88 

0.41 5 

0.03 

0.015 

0.46 

0.223 

C.  59 

0.419 

0.04 

0.020 

0.47 

0.228 

0.90 

0.423 

0.05 

0.025 

0.48 

0.232 

0.91 

0.428 

0.06 

0.030 

0.49 

0.237 

0.92 

0.432 

0.07 

0.035 

0.50 

0.242 

0.93 

0.437 

0.08 

0.040 

0.51 

0.246 

0.94 

0.441 
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0.045 

0.050 
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0.53 

0.251 
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0.95 

0.96 

0.445 

0.450 
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0.055 

0.54 

0.260 

0.97 

0.454 

0.12 

0.060 

O.55 

0.265 

0.98 

0 . 453 

0.13 

0.064 

O.56 

0.270 

0.99 

0.463 

0.14 

0.069 

0.57 

0.274 

1.00 

0.467 

0.15 

0.074 

0.52 

0.279 

1.01 

0.471 

0. 16 

0.079 

0.59 
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0.064 

0.60 
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0.18 
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0.61 
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0.484 
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0.489 
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1.07 

0.497 
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0.65 
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0.316 

1.09 

0.506 

0.24 

0.115 

0.67 

0.320 

•>  ^ 0 

0.510 

0.25 
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0.68 

0.325 

1.11 

0.514 

0.26 

0.128 

c!69 

0.329 

1 1 ~ 

0.519 

0.27 

0.133 

0.7  0 

C.334 

1 1 ~ 

i • - ^ 

0.523 

0.28 

0.137 

0.71 

0.333 

* l 

0 . 527 

0.29 

0.142 

0.72 

0 . 343 

0.531 

0.30 

0.147 

r-  ^ -o 
^ • ( J 

0 

u 0 . 

0.536 

0.31 

0.152 

0.7^ 

w ^ 

1.17 

0.540 

c.  32 

0.157 

0.75 

1 1 

0.544 

0.33 

O.lfl 

0 . 76 

r\  ~ £ 1 

-j  % j - 

1 1 G 

- • - y 

0 . 548 

0 I 1 

w . J~r 

0.166 

0.77 

0.365 

0. 552 

0.3  5 

0.171 

w • f Q 

0.37c 

1 . 21 

0.557 

0 . 36 

0.176 

0.79 

0.374 

1 . 22 

O.561 

0.37 

0.181 

0.50 

0.379 

1.23 

0.565 

0.3  8 

0.185 

0.81 

0.353 

1 . 24 

0.569 

0.39 

C.190 

0.82 

0.353 

1.25 

0.573 

0 . 40 

0.195 

0.53 

0.392 

1 . 26 

0.573 

0.41 

0.199 

C.S4 

0.397 

1.27 

0.532 

0.42 

0.204 

0.55 

0.401 

4 /~>  Q 

0.586 

V . ‘-rj 

0.209 

0.56 

0.  OCc 

1 . 29 

0. 590 

alues 

of  B as  a fun 

Colon  of  x 

for  Hurr 

.olet's  Algori 

approximating  Vg  ^,^(n)  at  100  points  between  0 and  1.0, 
and  Vg  0^(n)  points  between  0 and  1.5*  After 

18  iterations  the  capacity  is  bounded  below  by  .48711  and 
above  by  .48712,  and  the  optimal  initial  interval  length 
is  1.266/A.  These  are  exactly  the  values  found  by  Gallager 
using  other  methods.  Another  program  was  written  approx- 
imating vs  £x^(n)  and  Vg  ^ 0^(n)  only  two  points 

each:  VS^(0) <n)  and  ^ ( . 65) (n) ’ vS2(0,0)<n)  and 

Vg  (1  3 0^(n).  After  18  iterations,  this  program  bounded 
the  capacity  by  .48662  below  and  .48663  above,  and  found 
an  optimal  initial  interval  of  length  1.271/x.  These 
results  represent  errors  of  less  than  one-tenth  and  four- 
tenths  of  one  percent,  respectively. 

A program  analyzing  Humblet's  algorithm  using  the 
■pirst  grid  described  above  gave  a capacity  between  .48775 
and  .48776  with  an  optimal  initial  interval  of  1.275/a- 
The  program  which  analyzed  the  single  interval  algorithm 
with  a large  coarse  grid  (described  in  the  Appendix) 
converges  to  an  equivalent  stationary  policy  which  has 
the  same  capacity  and  initial  interval  length,  even  though 
the  grid  uses  only  1/6  as  many  points. 

This  insensitivity  of  the  method  to  grid  size  and 
density,  and  the  good  agreement  with  results  obtained  by 
other  methods  indicates  that  our  approximations  to  the 
value  functions  are  quite  accurate. 
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he  capacity  of 


3.5  Suggestions  for  Further  Research 

Many  fundamental  questions  about  t 
contention  resolution  algorithms  remain  unanswered.  Is 
the  best  single  interval  algorithm  the  best  first-come 
first-served?  Is  the  best  first-come  first-served  the 
best  possible  algorithm?  Kow  may  these  algorithms  be 
modified  for  the  case  where  not  all  users  know  the  out- 
come of  each  trial?  How  may  they  be  modified  when  the 
users  are  trying  to  access  more  than  one  facility? 


A.  APPENDIX 


Program  Notes 

In  this  appendix  we  include  listings  of  the  two  Fortran 
programs  that  were  used  to  calculate  the  value  function  and 
to  hound  the  capacity. 

The  first  program  calculates  approximations  to  Vg  (x)(n) 
and  Vg  y)(n)  using  a large,  coarse  grid.  The  grid  size 
was  determined  from  the  results  of  Humblet's  program,  de- 
scribed in  section  2.4.  For  his  algorithm  it  was  found 
that,  in  the  S^(x)  state,  for  values  of  x up  to  .75,  the 
optimal  policy  chooses  A=x.  For  larger  x,  A will  be  less 
than  x.  For  this  reason,  we  thought  it  unlikely  that  the 
single  interval  algorithm  would  cheese  A>x  for  x>.?8. 

For  safety,  however,  we  chose  the  grid  to  allow  A>x  for 
x up  to  .9 6.  We  expected  (incorrectly)  that  for  small  x, 

A would  be  much  larger  than  x,  and  as  x approached  ,?3,  A 

would  decrease  to  x.  Thus  we  chose  to  calculate  v~  , \(n) 

cs  2 v x , y / 

only  for  x and  y such  that  XTy<_.  96,  y> 0 . .since  Humblet's 

algorithm  found  an  optimal  initial  interval  length  larger 

than  Gallager's,  it  seemed  likely  that  the  SIA  would  use 

an  even  larger  T.  Hence,  we  calculate  v_  , n\(n)  for  x 

2 ^ ^ 9 ^ 

up  to  1.4?. 

The  program  was  initially  written  for  lx  (the  distance 
between  grid  points)  equal  to  . C6.  At  each  iteration  the 
program  would  calculate  value  functions  for  approximately 
200  states.  Changing  a single  card  in  the  program  (line  4) 
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would  change  lx  from  .06  to  .01.  This  was  never  acne,  since 
it  would  increase  the  number  of  states  from  200  to  5C00,  and 
the  program  would  have  been  too  costly  to  run.  The  program 
results,  however,  indicated  that  the  bounds  of  the  initial 
grid  were  much  larger  than  required.  The  program  was  re- 
written with  a small, fine  grid  using  approximately  the  same 
number  of  states  as  the  first  grid. 

The  second  program  uses  ix=.01.  Since  the  first  pro- 
gram showed  that  for  x f_.  12 , Afx,  we  restricted  the  grid 
to  allow  A>x  for  x up  to  .15-  This  extra  m.argin  was  included 
in  case  the  threshold  for  A might  change  due  to  the  finer 

discretization.  Values  for  v-  / ..(n)  were  calculated  only 

(x)  J 

for  xf . 79 . since  it  had  teen  shown  that  states  S„ (x)  for 
larger  x were  transient.  Since  the  first  program  showed 
that  the  largest  A>x  ever  used  is  .3  when  x=.06,  we  chose 
to  calculate  v,  , s (r.)  for  C < x<  . 1 5 and  C<y<.3.  Since 
the  largest  T ever  used  is  1.22,  we  calculated  v-  ^ Q)(n) 
for  x uo  to  1.29. 


The  second  program  also  eliminated  two  cost! 


S ? ( x , y ) was  found  always  less  than  or  equal  to  x,  so  the 
second  program  searched  a much  smaller  interval  for  5 . For 


v?  /-  3 is  positive  only 


very  xarge  y,  ccrrespcna- 


ir.g  to  transient  states,  sc  we  ret  the  second  program  cheese 
B(0,y)=0,  without  doing  a search. 

The  increment  for  the  controls  T,  A and  3,  was  chosen 
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to  be  Ax/10.  Hence,  the  finer  grid  gives  nore  accuracy  not 
only  cy  calculating  the  functions  at  nore  points,  but  also  by 
allowing  more  precise  determination  of  the  optimal  controls. 

Even  for  the  coarse  grid,  however,  it  is  not  practical 
to  test  all  the  possible  controls  for  each  state.  Hence 
the  search  for  the  optimal  control  is  carried  out  in  two 
parts.  First,  the  control  set  is  searched  over  a coarse 
grid,  with  increments  equal  to  Ax.  Then  a fine  search  is 
made,  using  increments  of  Ax/10,  over  an  interval  of  width 
2 Ax,  centered  at  the  optimal  control  found  from  the  coarse 
search.  Maximizing  a function  in  this  way  can  lead  to 
trouble  if  the  function  has  two  peaks  and  the  true  maximum 
lies  between  the  points  tested  in  the  coarse  search.  How- 
ever, the  difference  between  the  value  functions  at  a control 
T and  at  7 -lx  is  small  enough  *hat  any  error  of  this  type  is 
negligible . 
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